In manufacturing processes of semiconductor and magnetic disks, used are a charged particle beam measurement apparatus that irradiates charged particle beams (hereinafter, referred to as primary beams) such as electron beams and ion beams onto a sample, obtains signals of secondary charged particles (hereinafter, referred to as secondary beams) such as generated secondary electrons, and measures the shape and dimensions of a pattern formed on the sample, and a charged particle beam inspection apparatus that checks the presence or absence of defects. As such a conventional charged particle beam applied apparatus, there has been used a so-called SEM (Scanning Electron Microscope)-type apparatus that scans point-like primary beams on a sample. The SEM-type apparatus has characteristics of requiring a long time to obtain an image because the image is obtained by two-dimensionally scanning the primary beams, and the speed of processing a sample, namely, the inspection speed needs to be increased. In order to solve the problem, there has been proposed a multibeam-type charged particle beam applied apparatus in which plural beams are provided, or a multicolumn-type charged particle beam applied apparatus in which plural columns through which beams pass are provided (see Patent Documents 1 to 7). For example, Patent Document 1 discloses a multibeam-type electron beam inspection apparatus in which an electron beam emitted from a single electron gun is split into plural beams, and plural beams formed by being individually focused by array lenses are irradiated and scanned on a sample using a single optical element. In addition, Patent Document 2 discloses a multicolumn-type charged particle beam applied apparatus in which plural immersion lenses obtained by immersing an electric field or a magnetic field into plural apertures are produced and are arranged in the beam travelling direction, so that individual optical systems are arranged for plural beams.